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Abstract 

The purpose of this study is to investigate learning effects of on-line peer-support for generating critical questions and 
counter arguments in small group on-line discussion. The on-line peer-support includes descriptions of what to do and generic 
and domain specific examples of questions. About forty five students were recruited from an on-line introductory class on 
turfgrass management offered by a land-grant university. A field experimental time series design with control-group was 
employed. Data were collected from five discussion sessions, ten open-ended essay exams, and three multiple-choice exams 
through a semester. The results indicated that the peer -challenge guidance helped learners to generate more challenges 
(F=2.465, p < .05). But, it did not improve the quality of challenges. The increased quantity of challenges alone might be not 
sufficient to activate learners' reflection and critical thinking or to improve meaningful interactions. Consequently, it did not 
influence learning, performance in multiple-choice questions on memory and comprehension and open-ended essay questions. 

Background to the Problem 

On-line discussion has become one of the most popular Srategies used in on-line distance education as well as on-site 
education at the college level. For example, most on-line education support systems (e.g. WebCT) include various forms of 
computer supported communication functions, such as list-serve and bulletin board systems (Abowd, da Graca Pimentel, 
Ishiguro, Kerimbaev, & Guzdial, 1999; Hsi & Hoadley, 1997). In fact, many on-line instructors use these functions to encourage 
learners to be involved in discussions about target topics (Berge, 1997; 2000). Meaningful discussion helps learners to construct 
their own knowledge by providing several cognitive benefits such as articulation^ where learners articulate their understanding, 
perspectives, or opinions; cognitive conflicts ^ where learners reflect on new knowledge to justify or defend conflicting positions; 
and co-construction, where learners share and refine meaning with peers in a social context (Crook, 1994; Jonassen, Peck, & 
Wilson, 1999; Koschmann et al., 1996; Tao & Gunstone, 1999b). Although on-line discussion has been used largely with these 
expectations for learning benefits, the actual effects are unclear. In cases where simple question-answer cycles are employed, 
students do not become actively involved in critical thinking processes. These non-thoughtful interactions are not sufficient to 
promote active, knowledge construction. 

One possible reason for the lack of reflection during on-line discussions is that students do not know what to ask or how to 
ask questions (Miyake & Norman, 1 979; van der Meij, 1 990). Peer interactions can be initiated when learners raise thoughtful 
questions or provide critical feedback; however, in order to propose important questions or thoughtful feedback, question-askers 
need to have a certain level of domain knowledge and to activate metacognitive skills such as reflection, monitoring and 
evaluation (Dillon, 1986; Miyake & Norman, 1979; Palincsar & Brown, 1984; van der Meij, 1990; Wong 1985). Unfortunately, 
novice learners who begin to explore a new domain are often limited in those metacognitive skills, so they can neither ask the 
right questions nor generate productive feedback. It is a "metacognitive knowledge dilemma" (Land, 2000), whereby effective 
monitoring and reflection is linked to having some prior domain knowledge (Gamer & Alexander, 1989). 

This dilemma provides an essential need for developing on-line instructional strategies that can guide meaningful on-line 
discussion between or among peers (e.g. Abowd et al., 1999; Scardamalia & Bereiter, 1996a). Specifically, learners' generation 
of questions or feedback needs to be supported to lead meaningful discussions (Brown, 1989; King, 1994; King & Rosenshine, 
1993; Palincsar & Brown, 1984; Scardamalia & Bereiter, 1991; van der Meij, 1998). 

Purpose of the Study 

This Study develops a framework intended to overcome the metacognitive dilemma and to facilitate effective peer 
interactions in on-line discussion. This framework assumes that novice students who lack domain knowledge and cognitive skills 
can be supported in generating meaningful interactions at an early stage of learning (King & Rosenshine, 1993; Palincsar & 
Brown, 1984). The resulting questions and feedback in turn can enhance peers' metacognition, such as reflecting and monitoring, 
which allows them to refine and restructure their domain knowledge (Piaget, 1985; Webb & Palincsar, 1996). This framework for 
peer-challenge support has three assumptions. First, on-line support for students to generate meaningful challenges can increase 
the quality of students' questions and feedback by providing externalized support for metacognition (Palincsar & Brown, 1984). 
Second, in order to receive learning benefits from on-line discussion, such as articulation, cognitive conflicts, and co-construction 
of knowledge (Crook, 1994), this peer-challenging strategy should guide specific types of challenges that facilitate these learning 
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activities (Forman & Cazden, 1985). Thus, effective types of challenges should be questions that seek missing information from 
learners' explanations, counter arguments that contradict learner's opinions, and more systemic questions such as hypothetical 
questions that force learners to consider complex contexts and various perspectives. Third, once the quality of peer-generated 
challenges is increased, meaningful cycles of verbal interactions should be initiated. When learners receive critical, valuable, 
reasonable, and sophisticated questions or challenges from their peers, those challenges and interactions should magnify learner's 
cognitive dissonance and trigger a conscious cognitive process to re-construct and enhance existing understanding. Thus, the 
purpose of this study is to test this peer-challenge support framework by investigating the effects of on-line support for peer 
challenges on discussion activities and learning of college students. 

Research Questions 

Question 1 : Does the use of on-line guidance for generating effective peer-challenges affect students' on-line challenging 
behaviors such as types of challenges generated, clarity of challenges, and clarity of rationale in challenges during small group 
on-line discussion? 

Question 2: Does the use of on-line guidance for generating effective peer-challenges affect students' on-line discussion 
activities (frequency of interactions, threaded responses, and off-task interactions) during small group on-line discussion? 

Question 3: Does the use of on-line guidance for generating effective peer-challenges during small group on-line discussion 
affect students' performance in memory and comprehension? 

Question 4: Does the use of on-line guidance for generating effective peer-challenges affect changes in students' performance 
of open-ended essay questions during small group on-line discussion? 

Methods 

Participants 

About forty five students were recruited from an on-line introductory class on turfgrass management during the 2001 spring 
semester, which is regularly offered from a land-grant university in the northeastern United States. The audience for this on-line 
course has no restriction in their location and time. They can be full- or part- time students working toward either a degree or a 
certificate. In the 2001 spring semester, the majority of the participants were part-time male students who pursued a certificate for 
turfgrass management. The participants were randomly assigned to a small group of five to six members. Then those small 
groups were randomly assigned to either experimental or control group. 

Discussion tool and intervention 

An on-line discussion tool used for this class is called Collaboration and Negotiation Tool for Case-Based Learning (Conet-C 
version 1) and was designed by the authors. 

Guidelines for effective peer-challenges were embedded into the discussion tool to facilitate learners' generation of three 
different types of challenges to their peers' initial answers: clarification questions, counter arguments, and context- or 
perspective-oriented questions. 

Clarification questions are peer-generated questions seeking additional information from learners' initial answers for 
clarifying or elaborating the learners' ideas. These questions identify missing information, indicate unclear parts, and detect 
errors in learners' initial explanations on given topics. This type of peer-challenge could facilitate learners to elaborate/articulate 
their idea, explain these idea clearly,- and correct their partial misunderstanding (Koschmann et al., 1996; Tao & Gunstone, 
1999a). 

Counter arguments are peer-generated opinions expressing disagreements with learners' initial ideas. Unlike clarification 
questions, these opinions identify major differences between peers' and users’ understanding on given topics. This type of peer- 
challenge generates explicit ©gnitive conflicts which could encourage learners to justify their positions, reconstruct their 
misconceptions, and negotiate their understandings (Tao & Gunstone, 1999a). 

Context- /perspective-oriented questions are hypothetical questions changing critical factors in given problem situations or 
considering different perspectives on the problems. Unlike clarification questions or counter arguments, these challenges do not 
indicate any specific problem with learners' responses. Instead, these challenges could stimulate learners to systemically think 
about dynamic aspects of the problems beyond the levels of the assigned questions. This type of peer-challenge could facilitate 
learners to generate predications and explanations. 

Procedure 

Small groups of students were asked to answer the same set of five or six open-ended essay questions in five sessions of on- 
line discussion throughout the semester. At each session of discussion, students in each group were asked to answer their 
assigned question and post their answer within a week, so the initial answers were available for group members to review. Then, 
each student was required to ask questions or to provide different opinions at least two times to group members in each 
discussion session. At the same time, each student was asked to answer peers' questions or counter opinions about his/her initial 
answer. After completing each on-line discussion (approx. 1 week), each student revised his/her initial answer and submitted the 
final answer to the instructor. 

During the first two discussion sessions (pre-observation), there was no treatment given to the groups. After finishing the 
second discussion session, all students took a multiple-choice exam (pretest for memory and comprehension) in their bcal area 
administered by local librarians. In the third and forth discussion sessions (treatment observation), the guidance for effective 
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peer-challenges was presented to only the experimental group through the on-line discussion tool. After finishing the forth 
discussion session, another multiple-choice exam (post-test) was administered. In the fifth discussion (post-observation) no 
treatment was given to the groups. After this last discussion, the last multiple-choice exam (delayed and transfer test) was 
administered. 

During each discussion session, students' on-line verbal interactions and their initial and final answers for the given open- 
ended essay questions were recorded in a computer database for later analysis. 

Independent variables measured 

The followings are a list of independent variables measured. 

• Challenging activities 

• Challenge types: frequency of peer-generated clarification questions, counter arguments, and context -/perspective 
oriented questions. 

• Challenge clarity: how clearly peer-generated questions or disagreement points are described. 

• Challenge-rationale clarity: how clearly rationales for challenges are justified. 

• Discussion activities 

• Interactions: frequency of all postings 

• Threaded responses: the number of postings under one issue 

• Off-task discussions: frequency of off-^ask postings 

• Learning outcomes 

• Multiple-choice tests of memory and comprehension 

• Open-ended essay questions 

Results 

Effects on on-line challenging behaviors 

The frequency of three types of peer-generated challenges (clarification questions, counter arguments, and context - 
/perspective-oriented questions) were counted from the five discussion sessions respectively. According to the results of ANOVA 
repeated measures on the total frequency of three types of challengp generated during discussion sessions (see Table I), there was 
a statistically significant interaction effect for time and groups (F = 2.465, p < .05). The result indicates that the treatment group 
generated more challenges than the control group during and after treatment sessions (see figure I). In the post-hoc analysis, 
however, it failed to find specific time points and types of challenges that were attributable to the significant interaction effect. 



Table 1. Results of ANOVA on total frequency of three types of challenges 



Source 


Sum of 
Squares 


df 


Mean 

Square 


F 


Sig. 


Time* 


3.995 


4 


.999 


.466 


.761 


Time X Group 


21.137 


4 


5.284 


2.465 


.048 


Error (Time) 


317.229 


148 


2.143 







Figure 1. Total frequency of three types of challenges between control and treatment groups across five discussion sessions. 
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In order to obtain the scores for challenge clarity and challenge-rationale clarity, the challenges posted during discussion 
sessions were also evaluated by two judges according to rubrics. The results of ANOVA with repeated measures did not show 
any significant differences between two groups in both challenge clarity (F = .608, p. < .658) and challenge -rational clarity (F = 
.356, p. < .839) across discussion sessions. 

Effects on on-line discussion activities 

The frequency of all postings and off-task postings were counted throughout the five discussion sessions. Also, the average 
number of postings under each issues (threaded discussions) was calculated by dividing the number of issues into the number of 
all postings. The results of ANOVA with repeated measures of the on-line discussion activities did not show any significant 
differences between the two groups across all discussion sessions in the frequency of postings (F = .832, p. = .507) or off-task 
postings (F = 1.227, p < .302). In addition, the results of threaded discussions calculated from each small group did not show any 
consistent patterns of the curve in the average postings under each issue. This indicates that there are no clear differences 
between the two groups in threaded discussions. 

Effects on memory and comprehension tests 

The multiple-choice scores on memory and comprehension from pre-, post-, delayed-, and transfer tests were collected and 
analyzed by ANOVA repeated measures. The ANOVA results did not show any significant difference between the two groups in 
the scores across all tests (F = .060, p = .981). 

Effects on changes in open-ended essay questions 

The initial and final answers of students on the open-ended essay questions in each discussion session were collected and 
evaluated by two judges according to given rubrics. The ANOVA results did not show any significant difference between the two 
groups in the gain scores made from initial to final answers across all discussion sessions (F = 1.101, p = .358). 

Discussions 

In summary, the results indicated that the peer-challenge guidance helped learners to generate more challenges. But, this did 
not improve the quality of challenges. The increased quantity of challenges alone mi^t not be sufficient to activate learners’ 
reflection and critical thinking or to improve meaningful interactions. Consequently, this did not influence learning or 
performance on multiple-choice questions of memory and comprehension and open-ended essay questions on higher order 
thinking. 

One likely reason for the failure of the treatment to improve the quality of challenges may be that students did not frequently 
refer to the on-line guidance; it was not strongly emphasized by either the instructor or the interface of the on-line discussion tool. 

Instead, it was simply recommended by the instructor only twice during the treatment discussion session through the 
instructor’s announcement board. Thus, students may not have paid attention to the guidance during discussions. In addition, the 
interface design of the discussion tool may not have strongly encouraged students to look at and use the guidance; students in the 
treatment group were required to voluntarily seek this guidance by clicking on the guidance icon. Most students reported on a 
survey collected at the end of the semester that they referred to the guidance only one or two times during the treatment 
discussion sessions. 

Possible solutions to these limitations involve changes to the interface design and class administration, although these might 
generate additional problems. The guidance, for example, could be re-designed to open automatically whenever students open the 
discussion window. But, students might want more control without being “forced” to view the guidance for every posting. 
Another possible solution might be to provide a template for generating effective challenges that required them to go through all 
steps in order to post their challenges. If, however, we use more sophisticated interfaces for on-line guidance, then we may lose 
the feasibility of large-scale use of on-line strategies because instructors might find it difficult to incorporate such interface- 
dependent strategies into the generic discussion tools they use currently. Lastly, instructors might strongly emphasize the use of 
on-line guidance to students by sending messages to them more frequently. 

Although the current study did not find significant effects of using on-line guidance on discussion activities and learning, it 
showed very reasonable results indicating that the quality of challenges could be essential for meaningful discussion and learning. 
Further studies need to be focused on finding ways to improve the quality of challenges and need to be tested again to find 
possible learning effects of the on-line guidance. 
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